{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monte Carlo(MC) Estimation\n",
    "\n",
    "We will be using MC estimation on sample 4x4 grid world  \n",
    "\n",
    "![GridWorld](./images/gridworld.png \"Grid World\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monte Carlo Prediction for Estimation (\"first-visit\")\n",
    "\n",
    "Monte Carlo Prediction is carried out by sampling the trajectories over many episodes and using the rewards seen in samples as estimate for state values. The backup digram is given below. Pseudo code for the algorithm is given in Fig 4-2 in the text.\n",
    "\n",
    "![MC backup](./images/mc_backup.png \"MC Backup\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running in Colab/Kaggle\n",
    "\n",
    "If you are running this on Colab, please uncomment below cells and run this to install required dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Uncomment and execute this cell to install all the the dependencies if running in Google Colab or Kaggle\n",
    "\n",
    "## Uncomment and run for Colab\n",
    "# !pip install gymnasium==0.29.1 pygame==2.3.0 -q\n",
    "# !git clone https://github.com/nsanghi/drl-2ed\n",
    "# %cd /content/drl-2ed \n",
    "# %cd chapter4\n",
    "\n",
    "\n",
    "## Uncomment and run for Kaggle\n",
    "# !pip install gymnasium==0.29.1 pygame==2.3.0 -q\n",
    "# !git clone https://github.com/nsanghi/drl-2ed\n",
    "# %cd /content/drl-2ed \n",
    "# %cd chapter4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initial imports and environment setup\n",
    "import numpy as np\n",
    "import sys\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "sns.set()\n",
    "\n",
    "# create grid world environment\n",
    "from gridworld import GridWorldEnv\n",
    "size = 4\n",
    "env = GridWorldEnv(size=size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MC Policy Evaluation\n",
    "\n",
    "def mc_policy_eval(policy, env, discount_factor=1.0, episode_count=100):\n",
    "    \"\"\"\n",
    "    Evaluate a policy given an environment.\n",
    "\n",
    "    Args:\n",
    "        policy: [S, A]shaped matrix representing the policy. Random in our case\n",
    "        env: GridWorld env. In model free setup you have no access \n",
    "             transition dynamics of the environment.\n",
    "             use step(a) to take an action and receive a tuple of (\n",
    "             s', r, done, terminated, info)\n",
    "             env.nS is number of states in the environment.\n",
    "             env.nA is number of actions in the environment.\n",
    "        episode_count: Number of episodes:\n",
    "        discount_factor: Gamma discount factor.\n",
    "\n",
    "    Returns:\n",
    "        Vector of length env.nS representing the value function.\n",
    "    \"\"\"\n",
    "    # Start with (all 0) state value array and a visit count of zero\n",
    "    V = np.zeros(env.nS)\n",
    "    N = np.zeros(env.nS)\n",
    "    i = 0\n",
    "\n",
    "    # run multiple episodes\n",
    "    while i < episode_count:\n",
    "\n",
    "        # collect samples for one episode\n",
    "        episode_states = []\n",
    "        episode_returns = []\n",
    "        state, _ = env.reset()\n",
    "        episode_states.append(state)\n",
    "        while True:\n",
    "            action = np.random.choice(env.nA, p=policy[state])\n",
    "            (state, reward, done, _, _) = env.step(action)\n",
    "            episode_returns.append(reward)\n",
    "            if not done:\n",
    "                episode_states.append(state)\n",
    "            else:\n",
    "                break\n",
    "\n",
    "        # update state values\n",
    "        G = 0\n",
    "        count = len(episode_states)\n",
    "        for t in range(count-1, -1, -1):\n",
    "            s, r = episode_states[t], episode_returns[t]\n",
    "            G = discount_factor * G + r\n",
    "            if s not in episode_states[:t]:\n",
    "                N[s] += 1\n",
    "                V[s] = V[s] + 1/N[s] * (G-V[s])\n",
    "\n",
    "        i = i+1\n",
    "\n",
    "    return np.array(V)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Custom print to show state values inside the grid\n",
    "def grid_print(V, k=None):\n",
    "    ax = sns.heatmap(V.reshape(env.shape),\n",
    "                     annot=True, square=True,\n",
    "                     cbar=False, cmap='Blues',\n",
    "                     xticklabels=False, yticklabels=False)\n",
    "\n",
    "    if k:\n",
    "        ax.set(title=\"K = {0}\".format(k))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmZklEQVR4nO3ceVzU1f7H8beyDDuoYK6hAqJSGoqoebstLmmlqdVtuWWLmktqLt1Kb6l1q1u3Ms0t9yU1vGa5r1nWzV+7KCCIG+4o4C7rgPz+wE5NaE4JjDO+nv8kZ853Hh+PxIthZqhUXFxcLAAAJFV29AAAgKsHUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIDhbu9G7+iB5TkHfmPh3JccPcI15/aIEEePcE3x8nBz9AjXHC87vuLzSAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIDh7ugByoOnh7tG9b9bj9wTqyB/byXtOqIxk1bq8+92OHo0l3Hm5HFtXvWxDu5O1qE9qSrIy1WfMeMUFhX9u9flZp/VO4MfVfaZU/r7sFd0Y5vbKmZgJ/fDd99o7eqV2ha/RRkZx1StWrBiYlup74DBCg4JKbU/YWu8Jox7V6k7kuXr66v2HTup/6Ah8vHxdcD0zum7b7/RqpXLFb9lizKOHVW14GDFtmqtZwY9q5CQ6jZ7/2/z11q3drUSExKUtnePatSoqTUbPnfQ5FfGJaMw/dVH1b1dtCYu/EK7D2Tqsa6ttHRCf3V6erz+b+teR4/nEjKPHNCXyxYquGYd1bi+gQ7s3G7XdRsWzZI1P7+cp3M9E8eP1ZnTp9Wuw52qe32oDh86qMWLFurrrzZp/qJPVC34lzDs3JGigX2fUr36DTRk+PPKOHZMC+bN1sED+zVu0jQH/i2cy7ixb+v06dPq0LGTQkPr6dChg4pbOF9fbdqk/y5ZahPjNatWat3a1WrcuIlCqlf/nXu9+rlcFGKiQvW3TjEaMfZTjftwoyRpwcrv9NPif+r1Id10+xNjHTyha6jTIFKjZq2Qj3+AEr/ZpAVjR1/2mqMH9urb9cvU7v7HtWHRrAqY0nUMGf6CmkU3V+XKv/zEt03bW9SvV08tjluofgOfNeuTJ4yTf0CAJs+YKz8/P0lSzVq19caro/Tt/21W65vbVvj8zui550counkLmzNv+5db9NTjjypu4XwNfHaoWR80ZKhGvfIveXh4aOCAvtqza5cjRi4TLvecQvf2N6mwsEgzP9ls1vILCjVn2Tdq3ayB6lwX5LjhXIjF20c+/gF/6JoVsycoKvYW1WvctJymcl3RLWJsvjj9vBYQGKi0tF8e/Z47d07ff/eNOt3VxQRBku7q0lU+Pj7auGFthc3s7FrEtCx15i1iWiowMEh799r+xKF69evk4eFRkeOVG5eLQrNGdbXrQIbOZufZrP+YtE+S1DSyjgOmQsI3X2h/apI6P9rP0aO4jJycbOXm5CgoKMis7dm1U0WFhWrcJMpmr4eHpyIiGyl1R0oFT+lacrKzlZOTraAqVRw9SrlxuSjUCA7Q0cwzpdaPZpWs1QwJrOiRrnnW/HytnjdFf7nnAVWtXtPR47iMuAUfymq1qsOdnc1aVlamJKnaRZ58Dg4OUVZmRoXN54rmfzhXVqtVd3bqfPnNTsrlouBt8VC+tbDUel6+1dyOirVp6QIVFRXq9u6POnoUlxH/04+aMXWy2nfspJjY1mY9/8KT+J4enqWu8bRYlJ/Hk/x/1k8//qCpUyapY6fOatW6jaPHKTcu90Rzbr5VFo/Sfy2vCzHIvRAH2KfQalXuOdtHXr4BQars5mbX9Scy0vXV8jjd23uILN4+5TGiS7FaC3Tm9GmbtaAqVeX2q/Pel7ZXzw8bpLCwcI0c/S+bvRaLRZJUYC0odd8F+fmyeFnKYWrnZi0o0OnfnHmVqrZnnrZ3j4YOHqiw8AiNefW1ih6xQrlcFI5mnVGt6qV/RFQjuORJ0fTM06Vuw6Xt35mk6WOG2Kw9PynO7h8DbVg0SwFVg9WgSbROZKRLks6dOiFJyj5zSicy0hUUfF2pJ/SuVQlbt2pAnyds1j5dtUG1ateWJB07mq7B/XvLz89f702cKl9f2/cdBF94aerxzMxS952VlangEOd+uWR52Lo1Xr2f7Gmztnr9RtWuXfL849H0dPXr00t+/n6a9ME0+fr6XexuXIbLRSEh9ZBujYmQv6+XzZPNLW+oZ26H/WqGhqvXy+/arPkHVbX7+lNZGTp+9LD+M/ChUrctnfGeJGn0nJXy9vW/skFdRERkpCZ8MMNmrVpwsCTp9KlTGty/jwoKrJo2e9ZF37QWFh4hN3d3pSRvV/tfPddgtRZoV+oOtevYqXz/Ak4oMrKRps6YbbP2c1xPnTqpfk8/pQJrgebMWljqTWuuyOWi8Oln8Rr6eHv16tHWvE/B08NdPe9tre8T0nTo2CnHDuhkfPz8FdE05k9ff+dDvZR91vbR2dGDadoQN1O33vuwrm8YJU+L95WO6TICAgIV2/rmUuu5uTkaOrCvMjKOafL0Obo+tN5Fr/fz91dsq9Zau3qFnnq6v3kksWblCuXk5KhdhzvLc3ynFBAYqNZtSp95Tk6Onun3tDKOHdOM2fMUeokzdzUuF4UfkvZryfotenVQV4VU9dOeg1l6tEusQmtWU79XFjh6PJeycck8SVLGwTRJUvyX67VvR6Ikqd19JQ/HL/aeBK8LD7/rhDVSVOwtFTGq0xs14nltT0pUl249tC9tj/al7TG3+Xj76NY72puP+w0coj6PP6L+vXqq230PKOPYMS38cI5atWmrNm05b3uNfOE5JSUmqFuP+7R37x7t3furM/fx1R3tfjnznak7tOmLkl9rcfDAfp09d1bTPpgsSWoY2Ui33X5HxQ5/BVwuCpLU6+V5Gj3gHj18d6yqBPgoaddh9Xj2A23esufyF8NuG+Jm2nz84xerzZ9/jgLKxq7Ukt/btWLpJ1qx9BOb22rWrGUThUaNm2jC1JmaNG6sxr3zlnx8fNWl230aMHioYL/UHSVnvvSTJVr6yRKb22rVqm0ThZTkZE2aMN5mz88fd723u1NFoVJxcXGxPRu9oweW9yz4lYVzX3L0CNec2yNK/4we5cfLw75XsKHseNnxMICXfAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAMPd3o2DXhtUnnPgN2r7eTt6hGtOXsF5R49wTcnOL3L0CNec2kGel93DIwUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAACGu6MHKAt5Z05oz1crdGr/Tp08tFtF+blqO+B1BYffWGpvRmq8Dsf/TycP7NTZY4fkHRSsji/PcMDUzu3UiSytWxqnPanblbYrRXm5ORr51hQ1btriovsLrVatXjJfX29craxj6fL29VP9iMZ6atCLqhpyXQVP73yOZ2Xq47j5St6eoNSU7crNydG4KbMU3SK21N7CQqvmz56utauWKyvzmIJDrtNdXbrrkcd7yd3dJf6XrxDHszK1ZNF87UhKVOqOkjMfO3mWbmrRstTewkKrFsyZofWrlikrM0PBIdXVqUt3PdKzl9yc7Myda9pLOJdxWLs/XyLfkFoKqBmqk/t2XHLvoS1f6nD81wqq00BegVUrcErXkn5ov1YunqcateuqTr0w7U5JvOTewsJCvTN6qHYlJ+j2Tt1Ut364ss+d1Z7UJOXknFNVEYXLObA/TQvnzVSduqFqEBah7YnbLrn3tVEvatPG9bqrS3dFNo5SclKCZk6doGPH0vWPkWMqbmgnd3D/PsXNm6U6dUNVPyxCyb9z5m+MHqEvN65X5y7d1bBxlFKStmn21InKOJqu4U525i4RhcA6Yer8rwXy9PXXkW2b9cPvRKHxXY/ppr8NVGU3d30741WdST9QgZO6jnrhjTTlvxvk5x+o7/+3URNSRlxy79pPF2pH4ha9/M50hUVGVeCUriOyUZRWbNisgMBAbdq4XqNHDLvovpTkRH3x2Tr17NVPvfoOlCTde9+DCgwK0n8XzlOPBx5WWERkRY7utBo2aqKl679WQGCgvty4Xq8kDr/ovh3JSdr02To99lRfPXnhzLv2+JsCAqvo44/mqZuTnblLPKfg4eUjT19/u/Z6B1ZTZTeXaKFDefv4ys8/8LL7zp8/r/XLFinm5tsUFhmloqJC5eflVcCErsXH11cBgZc/74T4LZKkdh0626zf0aGziouL9fmGteUynyuy98wTt/4kSbr9t2feseTMN322rlzmKy98dUS5OnwgTSePZ6puvXDNHP+Gvv5slQoLrapbL1yP9humJs1iHD2iS7FaCyRJFovFZt3Ly1uStHNHcoXP5OqsBVZJksXL9swtFi9JznfmLvFIAVevY4dLfjy3dulH2pH4k54cPEJ9ho2S1Zqvt196VgfSdjl4QtdyfWg9SVJiQrzNesKF72YzMzMqeiSXV+fCmSdtsz3znx9BZGUeq+iRrgiPFFCu8vJyS/6bk6PXJs5XtQuvNGrSLEbP9eqhVYs/VP/nX3XkiC6l1c1/VY2atTRl/Dvy8vJSw0YlTzTPmPK+3NzcVZDPj+7KWuubb9F1NWvpg/fflcXLWw0bNdGO7Qma+cEEubm5Kz8/39Ej/iFOFYXzhVYV5JyzWbP4BahSZTcHTeT6Cq1WnTt72mYtILCKKrvZd+aeniUPqSOimpogSFJw9RpqGNVMu1ISym5YF2C1WnXmtO15B1WpIjc7z9tisejNsZM1ZuRwvfzCUEmSp6en+g4apvmzp8nb26fMZ3Z2VqtVZ8/YnnlgkP1n7mmx6I2xk/TqyOc05sWSM/fw9FTfgcO0wAnP3KmicGLfDm2e/E+btQ4vTZdPVV7SWF52pSTojRf626yNnbNUIdfVsuv6oGohkqTAoGqlbgsIrKr9e3Ze+ZAuJCkhXkP6P2WzFrd0nWrWqm33fdQPC9ecuKXat3ePzp49o3r1w2SxWDTpvf+oWXTp19hf67YnbNWwAbZnvvDTtarxR868QbhmffSp9qXt0bkzZxR64cwnj/uPmjZ3rufNnCoKAbXqq00/2x81WPyrOGiaa8P19SP0whsTbdYCq5T+An8pdeuFyc3dXSePl/5Z9qkTmQoIDLrSEV1KeESk3p043WatarXgP3w/lSpVUv2wcPPxt5u/0vnz59UitvUVz+hqwiIa6u0J02zW/vSZN7jImbd0rjN3qih4+vipesObHD3GNcXXP0A3RJd+16y9vH181azlzdr63WYdObhPterWk1TyqqRdyYm6/a7uZTSpa/APCFRMbJsyvc/8vDzNnDpR1YJD1L7jXWV6367APyBQLcrhzGdfOPM7nOzMnSoKvyd1wyJJ0tmjJa92OfjjFzqeVvJSsMgOD5p9p4+k6ej27yVJ2VnpKszLNtcG1qqvGlF//gvgtWbpRzMlSYf3p0mSvt64Wqnbt0qSuj3cy+z72+MDlLz1R/37xQHqeG/Jv8X6ZYvk6x+grg8+UaEzO7N5M6dKktLSdkuS1q9eocStJa946dmrr9k3esRwBYeEKLR+mHKyz2n1ik+VfviQ3hw7WT6+vhU/uBP7cFbJme/bu0eStGHNCiVuK3kvyGNP/XLmr4wcrmrB1VWvfgNlZ2dr7YpPdeTIIf177CSnO/NKxcXFxfZsfH5VannPckWWDet6ydvuHbvc/PnA9xsVHzf+ovvqtrxDzR8eUtaj/Sn3N67h6BEu67HOlw7oh2u+t/l43+4dips1UbtTElWpUmU1uamFHu41WDVqX1/eY9rt+mpX9xOCt8becMnbvvw+yfx54bxZWrPyUx1NPyKLxaKmN7XQk08/o4iGjSpiTLsV2felx6HuaFX696f97PPvfvnVLnEfztLalUvNmd/YrLmeePoZhV9lZ147yPOye1wmCq7GGaLgaq72KLgaZ4iCq7EnCrx5DQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAABGpeLi4mJ7Nu7NzCvvWfArReft+mdBGfL3cnf0CNeUb/cfd/QI15xuTWtcdg+PFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABjujh6gLJzIytTSxQuUmpyoXTuSlZubo7fen6GmzVuW2nv+/HmtWf6xVi/9WEcOH5CXl7fCGzbWw088rSY33lTxwzupE1mZWv7xQqWmJGr3hTN/Y/x0NY2++JmvXbFEa5YtVvrhg/Ly8lZYw8Z6qGcfNebM7XI8K1Mfx81X8vYEpaZsV25OjsZNmaXoFrGl9hYWWjV/9nStXbVcWZnHFBxyne7q0l2PPN5L7u4u8b98hThz8rg2r/pYB3Yn69CeVBXk5erpMeMUFhX9u9flZp/V24MfVfaZU/r7sFfUtM1tFTNwGXGJRwqHDuzT4gWzdTwzQ/XCIn5378xJYzXxnddVLyxcfQY+px4P9dShg/v1/MBeSk1OrKCJnd+hg/v08cKSMw9tEP67e2dNfk+T331d9RpEqPczw9Xtwcd0+OB+vTi4N2dupwP707Rw3kxlZWSowWU+x18b9aLmzJii5jGxGjTsRTWLbqGZUyfovf+8VkHTuobMIwe0adlCnTmRpRrXN7D7uvWLZsman1+Ok5Uvl/i2IbxRE/139VfyDwjU/77YoJSkbRfdV1RYqFVLF+svt3XQP15+w6zfcnsHPfm3u/XF+tWKbHJjRY3t1MIjm+ijlV/KPyBQX2/aoDdH/eOi+4oKC7Vm2WK1va29hr/0uln/y20d1fuhu7VpA2duj8hGUVqxYbMCAgO1aeN6jR4x7KL7UpIT9cVn69SzVz/16jtQknTvfQ8qMChI/104Tz0eeFhhEZEVObrTqtMgUqNnrZCPf4ASvtmkBWNHX/aaowf26tv1y9Tu/se1YdGsCpiy7LnEIwUfH1/5BwRedl9hUaHy8/NUpWpVm/WgKlVVuXJleVos5TWiy/mjZx5UpZrN+s9nbrF4ldeILsXH11cBgZc/74T4LZKkdh0626zf0aGziouL9fmGteUynyuyePvIxz/gD12zfPYERcXeovqNm5bTVOXPJR4p2Mti8VJkkxu1Yc1yNbqhmW5o2lzZ585q4Zxp8vMPUOeu9zt6RJfz85lvXLtcjaKaKqpZyZnHzS05805d73P0iC7Fai2QJFl+8w2Ol5e3JGnnjuQKn+lakfDNF9qfmqTh4+bpZOZRR4/zp11TUZCk50e9oX+Pel5vvzrSrNWoVUfvTpmrmrXrOHAy1zX8pdf11pgX9O5r/zRrNWrV0X8mzVGNWpx5Wbo+tJ4kKTEh3ubzOWHrT5KkzMwMR4zl8qz5+Vo1b4puuecBVa1ekyg4E28fX4XWD1PjG5qqWYtWOnniuBbPn6VXRwzR25NmKzCoiqNHdDnePr66vn6YGkWVnPmpE1lavGC2Xhs5VG9NnMWZl6FWN/9VNWrW0pTx78jLy0sNG0UpOSlBM6a8Lzc3dxXk5zl6RJf0xdIFKioq1O3dH3X0KFfMqaJgtVp19sxpm7XAoCpyc3Oz6/qiwkKNHPK0boyO0YChI8x6dEwr9Xushz5eOEe9Bgwt05mdndVq1bnfnHnAHzzzl4b21Y3RMeo35EWz3qxFKz3z+H365KO5erL/kLIc2alZrVadOW173kFV7D9vi8WiN8dO1piRw/XyCyWfy56enuo7aJjmz54mb2+fMp/Z2RVarco9d8ZmzTcgSJXtPPMTGen6anmc7u09RBYXOF+nikJK4la9MLi3zdqcxat1Xc3adl2fuO0n7du7W30GPWezXrtuqOqGNlBy4tayGtVlpCRt1chn+9iszVy0yu4zT9q2RfvTdqv3wOE267XrhqpOaAOlJG0tq1FdQlJCvIb0f8pmLW7pOtWsZd95S1L9sHDNiVuqfXv36OzZM6pXP0wWi0WT3vuPml3kfSTXuv07kzRtzBCbtRcmxalq9Zp2Xb9h0SwFVA1WWJNonchIlySdPXVCkpR95pROZKQrKPg6Va7sHK/rcaoo1A+P1BvvTbVZq1I12O7rT50o+Yc6X3S+1G2FRYUqKiq6sgFdUIPwSL029gObtT905iePSyp5A9tvFRVy5r8VHhGpdydOt1mrWs3+8/5ZpUqVVD/sl/ePfLv5K50/f14tYltf8YyupmZouHq//K7Nmn9Q1UvsLu1UVoaOHz2stwY+VOq2pTPekySNmbNS3r7+VzZoBXGqKPgHBCi65Z//pK5dN1SS9OXGtYpp3das705N0eED+3glzEX4+QfoppgrP/OvNq5Vi1a/OfOD+3RnF8781/wDAhUT26ZM7zM/L08zp05UteAQte94V5netyvw8fNXRNOYP319x4d6Kees7Y/8jh5M0/q4mbr13ocV2jBKnhbvKx2zwjhVFH7PR3OmSZL2p+2RJG1ct1LbE+IlSQ8/8bQkKaJRE0W3bK3P1ixXTvY5NY9toxPHs7T844/kabGo+9+c/0miihQ3t+Q72gP7Ss78i3WrlJywVZL00OMlP3IKj2yi6JjW2rh2hXKysxUd20Ynj2dqxZI4eVosuveBvztkdmc0b2bJo+S0tN2SpPWrVyhxa8nneM9efc2+0SOGKzgkRKH1w5STfU6rV3yq9MOH9ObYyfLx9a34wZ3YxiXzJEnHDqZJkrZ8uV77dpS8C7/dfT0l6aLvSfDy9ZMk1QlrpKjYWypi1DJTqbi4uNiejXszr+5XLXT+S7NL3rbm61/e4Zyfn6clH83Vl5+t1bH0I3L38FBU02j17POMwiIaVcSodik6b9c/i0Pd89ebLnnbyq+2mj/n5+fpk7h5+t/Gn8/cXU2aNtdjvQaowVV05v5eV/f3SLfG3nDJ2778Psn8eeG8WVqz8lMdTT8ii8Wipje10JNPP6OIhlfPWUvSt/uPO3qEy3rhgVsvedtbi7+85G17tsdr2pghV93vPurWtMZl97hMFFyNM0TB1VztUXA1zhAFV2NPFJzj6XAAQIUgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAw3O3dWMXXozznwG8UnS929AjXnPf+t9fRI1xTxo5839EjXHNy4ydedg+PFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAAhrujBygLP3z3jdatWalt8VuUkXFM1aoFq0XLVuo7YLCCQ0JK7U/YGq+J499V6o5k+fr6ql2HTuo/aIh8fHwdML1zysrM1OKPPtT2pATtSNmu3JwcTZg6W81jYkvtLbRaNW/2dK1ZuUyZGccUUv063d21ux59orfc3V3iU7Dc5Z4+od1fLdeJ/Tt18uBuFebn6q/PvKHqETeW2nt0xxYdiv9aJ/an6syxQ/IJCtZdo2c6YGrX4enhrlH979Yj98QqyN9bSbuOaMyklfr8ux2OHq3MucQjhUnjx2rLjz/o1jvaa9g/RqrDnZ21ccNa9Xy4h45nZdrs3ZmaokH9nlJeXq6eHfa8una/X8s+WayR/xjqoOmd04H9aZo/d6ayMjMUFt7wd/e+8vKLmjVtsprHtNKzz41Qs+gWmj5lgt59818VNK3zO5txWKkblyj39HEF1Az93b0Hf/pSB7Z8KQ9vX3kHVq2gCV3b9Fcf1eBH71Dc6h/03NtLVHT+vJZO6K+bb2rg6NHKnEt8m/bs8BfULLq5Klf+pXGtb75F/Xv31OJFC9XvmWfN+pQJ4+TvH6Ap0+fK189PklSzZm39+1+j9N03m9WqTdsKn98ZRTaO0prPNysgMEhffLZOLyVsvei+lO2J+nzDWj3Ru5/69B8kSep+/4MKCqqiuAVzdd+Djyg8IrICJ3dOVeqGqevrC+Xp669DWzfr2zlvXnLvDff0VIuHBqmym7u+nvaKzqQfqMBJXU9MVKj+1ilGI8Z+qnEfbpQkLVj5nX5a/E+9PqSbbn9irIMnLFsu8UghukWMTRB+XgsIDNS+vXvNWva5c/r+u290591dTBAk6a4uXeXj46PP1q+tsJmdna+vrwICgy67b1v8T5Kk9nd2tllvd2dnFRcXayNnbhcPLx95+vrbtdc7sJoqu7nE93tXhe7tb1JhYZFmfrLZrOUXFGrOsm/UulkD1bkuyHHDlQOXiMLF5ORkKzcnR0FVgsza7t07VVRYqMZNomz2enh4KiKykXamplTwlK6voMAqSbJYvGzWvbxKPk5N2V7hMwF/RLNGdbXrQIbOZufZrP+YtE+S1DSyjgOmKj8uG4W4BR/KarWqfcdfvkM9nlny/EJwcOknn4ODQ5SVmVFh810rrq9XT5KUuC3eZn1b/BZJUmYGZ46rW43gAB3NPFNq/WhWyVrNkMCKHqlcueRjzPifftTMaZPVrmMnxcS2Nuv5+fmSJA9Pz1LXeHpalJ+XX2EzXivatP2ratSspYnj3pbFy0uNGkdpe2KCpk0aLzc3d+Xn513+TgAH8rZ4KN9aWGo9L99qbnclThUFq7VAZ06ftlkLqlJVbm5u5uN9aXv1wvBBCgsL18hRtq9usVgsJfdTUFDqvgsK8mXxspTD1M7NnjP/PRaLRW+Pn6KXXxymf/5jiCTJ09NTAwYP19xZ0+Tt41PWIzu184VWFeScs1mz+AWoUmX7zhtlLzffKotH6S+VXhdikHshDq7CqaKQsG2rnunzhM3aJ6s2qFat2pKkY0fT9eyA3vLz89fYCVPl62v7voNqF96zkPWbl6n+vBYcUr18Bndiidu2alDfJ23WPl6xXjUvnLk9GoSFa/5/lylt7x6dPXNa9RuEyWLx0vtj31J085iyHtmpZaXt0FeTRtqsdX55hnyrXeegiXA064xqVS/9I6IawQGSpPTM06Vuc2ZOFYWIhpF6f8oMm7Vq1YIlSadPndLgAX1UUGDV1FmzLvqmtbCwCLm5uyslebvNcw1Wa4F2pe5Quw6dyvcv4ITCG0Zq3GTbM6964cz/iEqVKqlBWLj5+P++/krnz59XTKvWv3PVtSeodn3d0t/2Ea5XQBUHTQNJSkg9pFtjIuTv62XzZHPLG+qZ212JU0UhICBQsa1vLrWem5ujoYP6KjPjmCZPm6PrQ+td9Ho/f3+1jG2tdatW6Kk+/c0jiTUrVygnJ0d3dLizPMd3SgEBgWrZqk2Z3md+Xp5mTJmgasEhan/n3WV6387O08dP10Xe5Ogx8CuffhavoY+3V68ebc37FDw93NXz3tb6PiFNh46dcuyAZcyponApo0c+r+SkRHW5t4fS0vYoLW2Puc3Hx0e33t7efNxv4BA9/cQj6t+7p7r1eEAZGcf00Ydz1KpNW7Vpe4sjxndac2Z8IElK27tbkrRu9QolbC15VdETvfuZfS+/MEzVQkJUv0GYss9la9XyT3Tk8CG9PX5KqR/x4dJS1i+SJPNmtAM/fqHjacmSpMYdHzT7Th1JU3rS95Kk7Kx0WfOyzbWBteqr1g2lfxUJLu2HpP1asn6LXh3UVSFV/bTnYJYe7RKr0JrV1O+VBY4er8xVKi4uLrZn48mcovKe5U/rdld7HU0/ctHbatSspaWrP7NZ2xr/kyaNH6udO5Ll4+Ordh07qf+goVfVF6ii83b9szhU2xZRl7xt80+/vP9gwdyZWrV8qdLTD8tisahZdAv16vuMGkY2rogx7fbe//ZefpMDfTykyyVvu3/cCvPnfd99ph8/Gn/RfaEt71DLv18dv9Jl7Mj3HT2C3Sye7ho94B49dFdLVQnwUdKuw3pl8ip99o1zvbcpN37iZfe4RBRckTNEwdVc7VFwNc4UBVdhTxRc9s1rAIA/jigAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwKhUXFxc7OghAABXBx4pAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAAjP8HToHkQb8gxAgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a random policy\n",
    "random_policy = np.ones([env.nS, env.nA]) / env.nA\n",
    "\n",
    "# run mc policy prediction for 100 episodes for random policy\n",
    "V_pi = mc_policy_eval(\n",
    "    random_policy,\n",
    "    env,\n",
    "    discount_factor=1.0,\n",
    "    episode_count=100)\n",
    "\n",
    "# Print policy\n",
    "grid_print(V_pi.reshape(env.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAApcElEQVR4nO3ceVyU5f7/8bcLIMO+JYqKO2qWqSianjT3yhZbzvlWWrnklqKWdqx+lXVaTpvaoqlpx11LXEotLay0TdPUFFcUXFARUEFxEAbi9wd22TSoVMI4w+v5V1z3fc/jw+U583LmnrFCYWFhoQAAkFTR2QMAAK4eRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAABG5ZKe6N18WGnOgT94f9qTzh6h3OlUr6qzRyhXgnw8nD1CuePndfnXAbxSAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYFR29gClwdOjsp4bcpse6NlagX7eSkg8qnGTVuirDbudPZrbyD51QhtXL9XR/buVmrxXeedy9MDTbyqySbNLXnfubLamjukr6+lM9Yp9Vo1a31RGE7u2zRvX66vVnylh2xZlpB1XUEiIbmjZWo8MHKaQ0DCH83ds26oPJk3Qvj27ZPHxUYfO3dVvcKy8LRYnTO+aflr/oz7/bIV+2fKzjh8/rtCQUEW3jtGQYbEKDbvGnHcuJ0effrJEa7/+SvsS9yrHalWNWpG6+5771Ovef6pSpUpO/C3+PLeMwgcv9lavzs313vyvte9QuvrcEaNl7w5Rj4Fv64etSc4ezy2cOJai9Ss+UlB4hMJq1tGRxJ0lum7d4lmy5Z4r5encz/TJE3XmdJZu6tRNETVr6diRFH0at1Abvl+nKbMXKTgk1Jy7b+9uPTn8UdWqXUeDYkcrI/24Fs2fpSOHD+qVCe878bdwLe9OfEuns7LUuVt31aoVqSMpKfp44Tx9t26t5i1aotDzMU5JOaw3Xn1ZrWLa6ME+j8jH11frf/hO/335RW3f9oteePm/Tv5N/hy3i0L0tZH6Z49oPTV+qSbOWSNJmrdig35e9IxeHnmXbn5kvJMndA/hdRpo5JTF8vb11+6f1mlpCaKQfjhZW9YsV7u7euvbxbPKYEr3MTh2tJo2a6GKFS+849uqTTs9MbSfPolboL6Dhpv1D6e8I19/f705+UP5+PhKkqpWq64Jr76gTRt+UHTMjWU+vysaNfrfuqFFS7s9b9uuvQb2e0gfL5inocNHSpJCQ8O0cPEnqle/gTnvnvv+pReee0bLly3RgEFDVLNWZFmP/5e53T2FXl1uUH5+gWYs+d6s5ebla+YnP6pNs7qqUTXQecO5ES9vi7x9/f/UNV/OmayGLdupZtR1pTSV+7q+ebTdk9Nva37+ATp04MKr37Nns7X5p/Xq0v02EwRJ6nrLHfK2WLR2zeoym9nVtYhu5bDnLaJbKSAgQMlJF/Y8MCjILgi/ublTF0myO9cVuF0UmjWqqcRDaTpz1v4tik0JByRJ10fVcMJU2LVhrY4k7tTN9z/q7FHcRo7VqnM5VgUEBJm15H2JKijIV8NG19qd6+HhoXoNorR/L/fV/g6r9aysVqsCg4Iue+6JjHRJUmBQYClPdWW5XRTCQ/2Vmn7aYT01o2itWlhAWY9U7tnycvXV/Glq1eNuBYaFO3sct7Hko7my2Wzq0KW7WTt5ouiJKDg01OH84JAw80SFv2b+3Nmy2Wzq1v2WS55ns+VpwbzZioiooSbXutYrY7e7p+Dt5aFcW77D+rlcmzmOsvXj8oX6tSBfN975gLNHcRvbtmzSnBlT1KFzdzWPjjHrubm5kiQPD0+Hazw9Pc1x/HmbN23UB1Mmq2v3HmoV0+aS577+yktK2r9fEydNUeXKrvU061rTlkBOrk1eHo6/VpXzMcg5HweUTEG+TTnZZ+zWLP4BqlixZB+zy0xP1YaVi9T94WHyrOJdGiO6FZvNpjOns+zWAgKD7D7WeOhAssaNHaXadevr8afG2Z3r5eV1/nHyHB47Ly/PHMcFNluesrLs9zwoKNhuzw8kJ2nMqOGqV7+B/t+4ly75eLP/N0NLFy/S4GGxav+PDqUyc2lyuyikZpxW9Wsc3yIKDy26KXosPcvhGC4uZe9OzX9ltN3akAlzSvw20LeLZ8kvKES1GjdTZnqqJOls1klJkvV0ljLTUxUQco0qVHS7dzL/kp3bt2r0Y/3t1uYs+Vzh1SIkSWnHUzV25CD5+Prq5fGTZPHxsTs3OKToY5InMzIcHvvkifRiv9NQ3v2ydasG93/Ybu3Tz+NVPaJoz1NTj+mxQQPk6+undyZNkc8f9vz3ln+yVO9OfEv33PcvDRg4pFTnLi1uF4Vte1LUIbqB/Hyq2N1sbtW0tjmOkrsmsq7+b+xrdmu+AcElvv50RppOHT+q9x9/yOHY6pnvSJJGTV2qKr/7pEx5Vrd+lF57e5rdWnBw0f2B01mZGjtikGx5eXp96qxin+Dr1KuvSpUqa+/uHXb3Gmw2m/Yn7lGHTt0drinvGkZFadK0GXZrIefvyWRmntKwQQNky8vT+7Pm2n1p7Y+++XqNXhr3rG7u3FX/fua5Up25NLldFJbGb9Goh7uo/93tzPcUPD0q66E72+inbclKOZ7p3AFdjLePn+o0bfGXr7/pvr7KOWP/6iw95YDWxc1Um57/VET9JvLwqvJ3x3Qbfv7+atHa8f3qnByrnnl8qE6kp+mNSTNUo2bxn3v38fVTi1Yxil+9Ug/2HWReScR/vlw5Vqtu6tS1VOd3Rf7+AYpp4/jdjRyrVSOGDlZ62nFNmTFLtSJrX/QxNm/aqGeefELNW0Trpf++4fBRVlfidlHYmHBQi7/YrBeH36GwYF/tP5yh3re3VmS1EA1+YZ6zx3Mr3y8r2s/0lAOSpITv45WyN0GS1O6uByVJNaOaOlznZSl6VVCtbpQaRrcrg0ld33/HPaXdOxPUo2cvHTqQZPfdBG9vi9p16GR+7jt4uEYMfEhPDO2rW++8VxnpxxU3f7ZaxtyoVm3bO2N8l/T/nnpSOxK26Y5edys5ab+Sk/abYxaLRR3Pfw/h2NEjenzEY1KFCurctZviv1hl9zgNGkapQcOoMp3973C7KEhS/2dn6/mhPXX/ba0V5G9RQuIR3T1iir7fvP/yF6PE1sXNtPt529oL/2f4LQq4Mvbv3SNJWrViqVatWGp3rGp4dbsoNIhqotffmaYPJk3UlLffkMXHoh6391L/ISPKdGZXt3fPLknSp0uX6NOlS+yOVate3UThyJEjyj5T9GGM1175j8PjPDr4MZeKQoXCwsLCkpzo3XxYac+C33l/2pPOHqHc6VSvqrNHKFeCfPh4eFnz87r821qu+8YXAOCKIwoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMCoUFhYWluTE2ZsOl/Ys+J0bqgY5e4RyJ9jX09kjlCslfOrBFVQz2Ouy5/BKAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgFHZ2QNcCWdOndDG1Ut0dN9uHUveq7xzOer9zJuKbHLDJa87dzZb749+RNbTmbo79jk1jrmpbAZ2A6dOpGvlkgXatztB+/fu0rkcq8a9OVXX3hDtcO6vv/6q+JVL9MWKxUo9clhVqnirToNGurf3AEVd28wJ07uezRvXa83qldr+yxZlpB1XcEiIbmgZo76DhikkNMzh/B3btmrae+OVuGeXLD4+6ti5u/oPGSFvi8UJ07umoj3/TAnbivY8KCREzVu21iMDL7HnkyZo3/k979C5u/oPjnW5PXeLKJw8dlg/Lv9IweERCqtZR0cSd5bourVxM2XLPVfK07mno4cP6pOPZqlaRC3VqlNfe3duu+i5c6ZN1Iq4efpHl1vV/fb7ZD17Rl+uWKznH39U/3n7QzVo1LQMJ3dNH0yaoDOns3RTp26KqBmpY0dT9MmiBVr//VpNmxOn4JBQc+6+vbs1evgA1YqsqyEjxig9LVUfz5+llMMH9d+JU5z4W7iW6ZMn6vTpLHXo1E0RNWvp2JEUfRK3UOu/X6epsxc57PmY4Y+qVu06Ghw7Wunpx7Vo/iwdOXxQr05434m/xZ/nFlEIr9NQj09dIm9ff+3asE5LEl+87DVph5O1ec1yte/VR+viZpb+kG6mbsPG+nDJV/LzD9CP6+I1/sXio1BQkK8vlsepzU2dFTv2P2a9zU1dNKzPHfp2zedEoQSGjBijps1aqGLFC+/4tm7TTqOG9NWyRfPVb3CsWZ/x/tvy8/PX+Pc/lI+PrySparUIjX91nDZt+EHRMTeW+fyuaHDsaIc9b9WmnR4f2k/L4hao36DhZv3DKe/Iz99fb02+sOfh1apr/KsvuNyeu8U9BS9vi7x9/f/UNV/Omayo6HaqFcUT0l/hbfGRn3/AZc/Lz89XXm6uAoJC7NYDAoNVoWJFeXp6ldaIbuX65tF2T06/rfn5B+jQgWSzdvZstn7+ab06d+9pnpwkqdutd8jbYtE38avLbGZXd+k9TzJrF/b8Nrs973pL0Z6vXeNae+4WrxT+rF0b1ipl7w4NeuNDZaWnOnsct+blVUUNGjXVN6uXK6rJdWrUtLmsZ7MVN/cD+fr6qettdzt7RJeVY7XqXI5V/oGBZi15X6IKCvIV1biJ3bkeHh6q16CR9u3dVcZTupff9jwgIMis/bbnDRtda3du0Z5Had/e3WU95t9S7qJgy8tV/Lypan3LPQoMCycKZWD4Uy9pwktj9c6rz5q1qtUi9J+3P1TV6jWcOJlrW7xwjmw2m27u0sOsnTiRLkkKLuZGaEhoqLZv3Vxm87mjxR/Nlc1mU8cu3c3ayfN7HhIa6nB+cEiYEn5xrT0vd1H44dOF+rWgQO3ufMDZo5Qb3haLakbWU8Mm1+u65q2UefKEli2cqTeef0IvTpgu/9/9rQsls23LJs2eMUUdO3dX8+gYs56XmytJ8vTwdLjG09NLueeP48/btmWT5syYog5/2PPf9tSj2D33dLk9d6koFOTblJN9xm7N4h+gihUrlej6zPRUrV/5sXo8MlyeVbxLY0S3Y7PZlH0my27NPyBIlSqVbM8LCvL14pghurZZtPoPf9KsX98iRqMG3KdPP56j3o/GXuIRyhebzaYzp+33OyDQfr8PHUjSc/8eqTr16uuJp1+wO9fTq+geTZ4tz+Gx8/Jy5eXFPZw/KtmeJ+v5saNUu259PfHUOLtzf9tTW7F7nudye+5SUUjZu0NzXx5tt/bYxLkKDAsv0fXr4mbKLyhUkY2bKfP820bZWackSdYzmcpMT1VAyDWqUNEt7r9fEXt3/KJxowfZrU2au1zXhFcv0fU7t23W4QP79fCQx+3Wq9WopRq16mh3wtYrNapb2LFtq554rJ/d2rwlqxRePUKSlHY8Vf8eMUi+vr56ZfxkWXx87M4NCSl62+hkRrrDY5/IyCj28/Xl3Y7tWzX6sf52a3OXfK7war/b85GD5OPrq1fGT3LY8+Dze34iI8PhsU+eSHe5PXepKFwTWU8PPPWa3ZpvQHCJr886kaZTx49o0qg+DsdW/e8dSdIT05apyu8+QVDeRdZrqGdfm2y3FhgccpGzHWWdOilJ+rXgV4dj+fn5+rWg4O8N6GbqNWio19+ZZrf22+fhs7Iy9e/YgcrLy9PbU2cX+2RTu159VapUWXt27VTH391rsNls2p+4Wx06d3e4pryrVz9Kr739hz0PvrDnY0cMki0vT29MnXXJPd+7e4fdvYaiPd+jDp1ca89dKgrePn6q07TlX76+4319ZT1z2m4t/XCy1sbNVNue/1JEgyby8Kryd8d0K75+/rq+ZczlT7yIajUiJUnff7NazVtf+Kx2UuIuHU05qC639vrbM7oTP/8AtWzd1mE9J8eqp0cNVUZ6mt6aPEM1akUWe72vr59atIrRmtUr1KffIPO32i8/X64cq1UdOnUr1fldkZ+/v1q2buOwnpNj1TOPF+35m5NmqEbNy+35SvXue2HP48/v+U2dupbq/FeaS0XhUr5bOleSlH7koCRp+3fxOrwnQZLUvldvSVLNqOscrqtiKfoDrFY3SlHR7cpiVLexeO50SdLhg0Wf2V4b/5l5O+ie3gMkSfUaNtb1LWO09osVyrGeVbOWbXTqZIY+X/aRPD29dNs93PAviVeeH6vdO7erx+29dDA5SQeTL3xO3ttiUfsOnc3P/QbHKnZgH40a0lc977pX6WmpWrRgtqJjblTrtu2dMb5LenXcU9q9M0E9evbSoQNJdt9N8Pa2qF2HTubnfoOHK3bgQ3p8aF/ddue9Sk8/rrj5s9XSBfe8QmFhYWFJTpy96XBpz/K3vPxgl4see2Ze/EWPHdy5VXNfHn3V/dtHN1S9+j+Rc1+Xi79qWxT/s/nv3NxzWr5ojr7/+gulpR5R5coeanxdc/3rkSGqUz+qLEYtkWBfx0+PXC0euKu7jqceLfZY1fDqmr/M/gtS27du1geTJihx7y5ZLD7q0LmbBgwZ6fB+uDOV8KnHaR7s1eOSez5v6Sq7te2/bNb0SRPP/3tTFnXoVPTvTV1Ne14z+PI3vd0mCu7GFaLgbq7mKLijqz0K7qgkUeBjNgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwKhQWFhaW5MRjWXmlPQt+59cS/angSjqZzf/Gy9KW1FPOHqHceSi65mXP4ZUCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAo7KzB7gSfv5pveJXr9T2rVuUnnZcwSEhah4do/6DhykkNMzu3I3rf9DXX67Szh3bdehAksKqhuujT1Y7aXLXtXnjesWvWqmEX+z3vO8gxz2XpIRtWzXt3fFK3LNLFh8fdezSXQOGjJC3xeKE6V3PqRPpWrlkgRJ3JWj/3l06l2PVuLemqukN0Q7n/vrrr/py5RJ9uXyxUo8cllcVb9Vp0Ej39hmgRtc2c8L0runMqRPauHqJju7brWPJe5V3Lke9n3lTkU1uuOR1585m6/3Rj8h6OlN3xz6nxjE3lc3AV4hbvFKY+t4Ebf15o/7RsZOGPzFWnbrdom/WrNaA3vfpREaG3bnxq1cq/ovP5OvrW+yTF0pm2nsT9MvmjWrfsZOGPTFWN3e9Rd/Er9bAPvfp5An7Pd+3d7dGDxugc7nnNGTkGN16591asSxOLzz9uJOmdz1HDh/UsoWzdDIjXbXq1L/kuXOmTtQHE19VrTr19fCQx3X7fb11LOWgnh/1qBJ3J5TRxK7v5LHD+nH5RzpzKkNhNeuU+Lq1cTNlyz1XipOVLrd4pfDYyDG67oYWqljxQuNat2mnEYP7aumi+RowJNasPzp0hMY8M06VK3to7KjHlJy0zxkju7whI8foumb2e96qbTuNOr/n/Qdf2PPpk9+Wn5+/Jkz+UD6+vpKk8GoReuuVcdq4/ge1anNjmc/vauo1bKz/Lf1Kfv4B+nFtvN56cVux5xUU5Gv18ji1uamzYp/6j1lv26GLHut9h76N/1wNGjUtq7FdWnidhnp86hJ5+/pr14Z1WpL44mWvSTucrM1rlqt9rz5aFzez9IcsBW7xSqFZi2i7J6ff1vz9A3ToQLLdemjYNapc2aMsx3NLzZoXs+fNz+958oU9P5udrZ9/Wq8uPXqaIEhSt1vvkLfForVreOuuJLwtPvLzD7jsefn5+crLzVVgUIjdekBgsCpWrChPL6/SGtHteHlb5O3r/6eu+XLOZEVFt1OtKNcNr1u8UiiO1WpVTo5VAYGBzh6l3MgpZs+T9yeqoCBfUY2b2J3r4eGh+g0aKXHPrjKe0r15eVVRg8ZN9c3q5WrY5Do1vq65zmZnK27uB/Lx9VPX2+529ohua9eGtUrZu0OD3vhQWempzh7nL3PbKMQtmCObzaabu/Rw9ijlRtzCoj3v2PXCnp/ISJckBYc43r8JDg3V9q2by2y+8iL2qZc04T9j9c6rz5q1qtUi9NI7H6pq9RpOnMx92fJyFT9vqlrfco8Cw8KJwtXml82bNGv6FN3cpbtatIpx9jjlwi9bNmn29Cnq2KW7WkRf2PPc3FxJkoenp8M1np5e5jiuHG9vi2rUrqeGTa7XdS1aKfPkCS1dMFOvP/eE/jNxuvwDgpw9otv54dOF+rWgQO3ufMDZo/xtLhUFm82m01lZdmuBQUGqVKmS+fnggSQ9++RI1alXX2OeeaGsR3Q7NptNZ07b73lAoP2eHzqQpOfP7/nop+333Ov8e9i2vDyHx87LyzXHUcRmsyn7jP1++wfY7/elFBTk68Unh+jaZtHqP/xJs359ixiN6n+fPvlojvoMjL3EI5Q/Bfk25WSfsVuz+AeoYsWS7XlmeqrWr/xYPR4ZLs8q3qUxYplyqSgkbNuqUUP62a0tWLZK1apHSJLSjqdqzPBB8vH11WsTJ8vi4+OMMd3Kjm1b9fhQ+z2fv3SVwn+350/GFu35qxMc9/y3j/2ePJHu8NgnMzL4WPAf7Nnxi8Y9MchubfK85bomvHqJrt+5bbMOJe/Xw4PtP+5brUYtRdSqoz07tl6pUd1Gyt4dmvvyaLu1xybOVWBYeImuXxc3U35BoYps3EyZ5982ys46JUmynslUZnqqAkKuUYWKrvG5HpeKQv0GDfXme9Ps1oJDQiVJWZmZGj18oGy2PI2fNJsnmyukXoOGeuPdi+x5VqaejB0oW16e3pxW/J7XrldflSpV1p5dO9Xxd/d3bDab9iXuVsfO3Uv3F3Axtes11HOvT7ZbCwwOucjZjrJOnZRU9AW2PyrIz1dBQcHfG9ANXRNZTw889Zrdmm9AcImvzzqRplPHj2jSqD4Ox1b97x1J0hPTlqmKj6/D8auRS0XBzz9A0a3bOqzn5Fg1dtRQZaSnacLkGapRK9IJ07knP/8AtbzInj81aqgy0tI0/hJ77uvrp5atYxS/aoX69BtkXkl8+fly5Vit6tC5W6nO72p8/fx1fcu/fh+sWo2iP4fvv16t5q0vfP8jae8uHU05qC639frbM7obbx8/1Wna8i9f3/G+vrKeOW23ln44WWvjZqptz38pokETeXhV+btjlhmXisLFvPTsWO3asV233t5LBw8k6eCBJHPM29uif3TsbH7en7hH36/7RpJ0JOWQzmaf0ewZUyVJ9RtG6cZ/dCzL0V3WK8+N1e4d23VLcXtusah9hwt73m9wrIY/2kcjh/RVz7vuVXpaqhbNn63omBvVum17Z4zvkuLmTpckHT6/1+u+/Ey7E7ZKku7tPUBS0Zfcrm8Zo2++WCGr9ayaRbdR5okMfb7sI3l6eum2e1z/RmhZ+m7pXElS+pGDkqTt38Xr8J6ib4W379VbklQz6jqH66pYiv7yU61ulKKi25XFqFdMhcLCwsKSnHgsy/FG4dXiX3d21/FjR4s9VrVadbt/2+jzFcv02ovPFntu99vu0FPPv1wqM/5Zv5boT8V57r/r0nu+YJn9l9K2b92saZMmFP3bRxYfdejcTY8OHXlV3fc5mX31/m9cku7tfPG/zcat+dn8d27uOX368Rx9//UXSks9osqVPdT4uub6v75DVKd+VFmMWiJbUk85e4TLevnBLhc99sy8+IseO7hzq+a+PPqq+7ePHoquedlz3CIK7uhqj4I7utqj4G5cIQrupiRRcI3b4QCAMkEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGBUKCwsLCzJiefyS3sU/N7J7Dxnj1DurEpMdfYI5cpjA1939gjlTs6W9y57Dq8UAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAABGZWcPcCVsWP+jVq74VFs2b1ba8VSFhIaqdUwbPTZ8hMLCrrE7d/q0Kfrm66+UcviQzp49q/DwavrHTR00YNAQBQcHO+k3cD0/b1yv+FUrlfDLFqWnHVdwSIiaR8eo36BhCgkNczg/YdtWTX13vBL37JLFx0cdu3TXo0NGyNticcL0rif71AltWr1Ux/bvVmryXuWdy9H9T7+pWk2aXfK6c2ez9cGYvrKeztSdsc+qUeubymhi9+LpUVnPDblND/RsrUA/byUkHtW4SSv01Ybdzh7tinOLKEwc/4aysrLUtVsPRUbWVkrKYS2cP1frvvlGHy9eptCwC09SO3fsUFSjRupxy62y+PgoOSlJS+I+1rfr1uqjxctk4UmqRKa9N0FnTmepQ+duqlEzUkePpGjZogX68bu1mj43TsEhoebcfXt364lhAxRZu66Gjhyj9LRUfTRvlo4cPqjXJk5x4m/hOk4eS9GGFR8pKDxCoTXr6GjizhJd993iWbLlnivl6dzfBy/2Vq/OzfXe/K+171C6+twRo2XvDlGPgW/rh61Jzh7vinKLKIx+8ik1b9FSFSteeDesXft/qN/DvbVw/lwNGzHKrI9/+12H65s1u0FPjIrV2m++1i233lYmM7u6oSPH6LpmLez2vHXbdho5uK+WLpqv/oNjzfr0yW/Lz89fEyZ/KB9fX0lSeLUIvfnKOG1c/4NatbmxzOd3NVXrNFDslMXy9vXX7p/W6ZMSRCH9cLK2rFmuG+/qre8WzyqDKd1T9LWR+mePaD01fqkmzlkjSZq3YoN+XvSMXh55l25+ZLyTJ7yy3OKeQsvoVnZPTr+tBQQEKinp8hWvHhEhSTpz5nSpzOeOmjWPdtjzZs2j5e8foIPJyWbtbHa2Nv20Xl169DRBkKRut94hb4tF36xZXWYzuzIvb4u8ff3/1DVr5kxWg5btVDPqulKaqnzo1eUG5ecXaMaS781abl6+Zn7yo9o0q6saVQOdN1wpcIsoFMd69qys1rMKDApyOFZYWKhTp04qIz1dm3/epNdeeUmVKlVSq1atnTCp+8ixWpWTY1VAYKBZS9qfqIKCfEU1bmJ3roeHh+o3aKTEPbvKeMryYfeGtTqSuFM33/+os0dxec0a1VTioTSdOWv/NtymhAOSpOujajhhqtLjFm8fFWfunFmy2Wzq3uMWh2MnMjLUuWN783PV8HC9+vqbqlO3XlmO6HbiFs6RzWbTzV17mLWTGemSpJAQx5vPwaGh2r51c5nNV17Y8nL19fxpiu5xtwLCwpWVftzZI7m08FB/paY7vouQmlG0Vi0soKxHKlVuGYWfN23U1PcnqVuPWxTTpq3D8YCAAE2d/j/l5uZq966dWhP/paxWqxMmdR+/bNmkWdOnqGOX7moRHWPWc3NzJUkenp4O13h6epnjuHLWL1+oXwvy1fbOB5w9ilvw9vJQri3fYf1crs0cdycuFQVbXp6ysrLs1oKCg1WpUiXzc3LSfo2KHaZ69Rto3IsvFfs4Hp6eatO26OZmh443K6ZNWz3c+34FB4eoQ8ebS+8XcEE2m01nTtvveUBgkN2eHzqQpOeeHKk69eprzNMv2J3r5eVV9Dh5eQ6PnZeXa46jSEG+TTnZZ+zWLP4Bqlix0kWusJeVnqqfVi5S14eHybOKd2mMWO7k5Nrk5eH4VFnlfAxyzsfBXbhUFLZu3aIBfR+yW/vsizWKiCh6Ty/12DENfrS/fP18NWnKNPn4+Bb3MA5uaN5CYWFh+mzFcqLwBzu2bdWoof3s1hYsXaXw6kU359OOp2pM7CD5+PrqvxMmy+LjY3du8PnvLJw4ke7w2CczMor9TkN5dmTvTi14ZbTd2uAJcxQQFl6i679dPEt+QSGq1biZstJTJUnZWSclSTmns5SVnir/kGtUoaLb3k684lIzTqv6NY5vEYWHFt34P5ae5XDMlblUFKKiGmnq9P/ZrYWef1LJzDylwQP7Kc+Wp5kfznf40trl5ObmKfsPf0ODVK9BQ7357jS7td++g5CVlakxsQNly8vTW9NmF/sEX6defVWqVFl7du3UzV0u3Guw2Wzal7hbHTt3L91fwMVcE1lX/xr7mt2aT0DJv1R5OiNNp44f1dTHH3I49sXMdyRJI6YuVZUS/oUJ0rY9KeoQ3UB+PlXsbja3alrbHHcnLhUF/4AA87bP71mtVj02eKDSjh/X9P/NVmRk7WKvt1qtqlChgry97V9Wx3+xWqdPZ6nJtU1LY2yX5ucfoJatHe/L5ORYNXbUUGWkpWn85BmqUSuy2Ot9ff3UsnWM4let0EP9BplXEl98vlw5Vqs6du5WqvO7mio+fqrdtMVfvv6m+/rKesb+b64ZKQf0bdxMxfT8p6rXbyIPryp/d8xyZWn8Fo16uIv6393OfE/B06OyHrqzjX7alqyU45nOHfAKc6koXMzT/x6thO3bdNfd9ygpab+SkvabYxaLjzp17iJJOnTwoAYNeETde9yq2nXrqmKFitqxI0GfrfhU1SMi9GAfx79doXgvPzdWu3ds1y2399KhA0k6dODC90G8LRa179DZ/Nx/cKyGPdpHI4f0Vc+77lV6Wqo+nj9b0TE3qnXb9sU9PIrxw7J5koqe5CUp4ft4pexNkCTdeNeDkqQaUY5/saliOf+FwbpRahjdrgwmdS8bEw5q8Reb9eLwOxQW7Kv9hzPU+/bWiqwWosEvzHP2eFecW0Rhz+6if39k2ZLFWrZksd2x6tUjTBSqhldV567d9dNP6/XpJ8uUn29TteoR+r/7H9SAQYMVGOj4nQYUb1/iHknS58uX6vPlS+2OVa1W3S4KDRs10VvvfqCpkyZo0sTXZbH46Nbbe+nRoSPLcmSX923cTLuft69dZf77tyigdPR/draeH9pT99/WWkH+FiUkHtHdI6bo+837L3+xi6lQWFhYWJITzzl+Igul6GS246d1ULpWJaY6e4Ry5bGBrzt7hHInZ8t7lz2HjyAAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACjQmFhYaGzhwAAXB14pQAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADD+P3l4MLitQFB6AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# run mc policy prediction for 10000 episodes for random policy\n",
    "V_pi = mc_policy_eval(\n",
    "    random_policy, env,\n",
    "    discount_factor=1.0,\n",
    "    episode_count=10000)\n",
    "\n",
    "# Print policy\n",
    "grid_print(V_pi.reshape(env.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAnuUlEQVR4nO3ceVxVdf7H8bcLIJfLjoWiormgppmKoOmkE6Q2WZNtMzXapOaC41raZDNju02NqW2mZuXeIoqNNmlqpb9KTXPLHQUVUARUULwIF+L3B/rV23WhEq73+nr+ld9zzn18+E5zX3HuPVYpLS0tFQAAkqq6egAAwNWDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMCoXt4TfVsPqcg58DNT3v27q0e45sQ3ut7VI1xTQvy8XT3CNcfX6/Ln8JsCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAo7qrB6gI3l7VNTbhTj3cI0ZB/r7alnxIz769RF+u2+Xq0TxG/vGjWr9soTL27lJm6h4VnS7QX/4xXpHNb77kdadP5WvKqEdlO5GrnsPGqlnsrZUzsJvbuH6tViz9TNu2bFJ21hGFhIaqdXSs+gwcotCwmk7nb9u6WdPenKDk3Ttl8fNTl/hueixhuHwtFhdM757WrV2j/y35rzZt2qisI5kKDQtTTEx7DR46XDVrXmfOKygo0KeLFurrL1dqb/Ie2WynVLdepO67/0Hd98CfVK1aNRf+FL+cR0bh3ed7qWdca7017yvtPZit3nfHatGbCeo+4HV9tznF1eN5hKOH07Rm8ccKCY9QzboNlJG8o1zXrU6cIXvh6QqezvNMe2uiTp7IU+e4roqoG6nDGelaNP9Drflmld6dk6iQ0DBz7t49uzRqyGOqV/8GJYwYreysTH0yd6Yy0g7o35OmuPCncC+vT/iP8k7k6fau3VWvXn2lp6fp4w/naPWqr/XxgkUKOxPjjPQ0vTLuBcXEdlCvRx6V1WrVd99+o3EvPqetW7foxXGvuPgn+WU8LgrRN0bqwe7RGjMhSZNmr5QkzV2yTj/M/4deGnGPfv/oBBdP6BnCGzTRyKkL5WsN0M51q5WU/Pxlr8lKS9XGlYvVqWdvrU6cUfFDepCEEaPVslUbVa167o5vuw4dNXJQHyXNn6d+g4aZ9emTX5e/f4AmTn5fflarJCm8VoReG/es1q/9Tu3a31Lp87ujJ54co9Zt2jrsecdOv1O/R3vpo3lzNGTYSElSaFiY5ictVqNGjc159z/4Zz3zzzH6dNFCDRg0WPXqRVb6/L+Wx32m0DP+ZhUXl+i9hd+atcKiYs34dI3at7pBda4Pct1wHsTH1yJfa8Avumb57MlqEt1RdaNaVNBUnqtV62iHN6ezawEBgTqYmmrWTuXn64fv1yq+ew8TBEnq+oe75WuxaNXKZZU2s7trG93Oac/bRrdTYGCQUlPO3XEIDg5xCMJZt8XfLklKTdlXsYNeYR4XhVZN6yr5YJZOnnK8RbFh235J0k1RdVwwFXauW6WMPdt120MDXD2Kxyiw2VRQYFNgUJBZS92XrJKSYkU1a+5wrpeXlxo1bqrk3TsreUrPYrOdks12SkHBwZc9NycnR5IUFHT5c68mHheF8LAAZWafcFrPzClbq1UzsLJHuubZiwq1cu5Uxdxxn4Jqhrt6HI+R+NFs2e12dbm9u1k7mpMtSQoJdf7wOSQszBzHrzN39kzZ7XZ1637HJc+z24s0d/ZMRdSpoxtbtKyk6a4Mj/tMwdfHS4X2Yqf104V2cxyVa81/P9JPJSW65Y8Pu3oUj7Fl0wbNmj5FXeK7qU10rFkvLCyUJHl5eztd4+3tY47jl/thw3pNfedtde12h2JiO1zy3JdfekEp+/bqzcnTVL26e73Nute05VBQaJePl/OPVeNMDArOxAHlU1JsV0H+SYc1S0CgqlYt39fscrMztfazT9Tt0aHyruFbESN6FLvdrpMn8hzWAoOCHb7WeHB/ip55coQaNGykUU8/53Cuj49P2esUFTm9dlFRoTmOc+z2IuXlOe55cHCIw56npuzT48OHqGGjxnrm+Rcv+Xoz3p+uhYmf6G9Dh+t3t3aukJkrksdFITPnhGpf53yLKDys7EPRw9l5Tsdwcel7tmvuS6Mc1gZPmlPu20CrE2fIPzhMkc1aKTc7U5KUn3dckmQ7mavc7EwFhl6nKlU97k7mr7J962Y9Privw9q8pKUKrx0hSco6kqknhw2Un9WqlydOlsXPz+Hcs88sHDvqfJvoWE7OBZ9puNZt3rRJ/fs+4rD22bKViogo+/wx8/BhJQzoJ6u/VW+9M01+ftYLvYwk6dNFC/X6xPF64ME/q//AwRU6d0XxuChs3Z2uztGN5e9Xw+HD5nYt6pvjKL/rIhvqoTGO37O2BoaU+/oTR7N0/EiGJo/s7XRs2QdvSJIen7ZINS7xf7RrScPGTfSfN6c5rJ19BiEvL1dPDhsge1GRxk+bdcE3+PoNG6lateravXOHusSf+6zBbrdrb/IudYnrVrE/gBuKimqqKe9+4LB29hmE3NzjShjQV0VFRfrgvXkOD6393FdfrtDzz/xTcfFdNeafz1TozBXJ46KQtGKTRv41Xv3u7WieU/D2qq5H/the329NVfqRXNcO6GZ8/fzVoEXbX3195wf6yHbS8YP/7LRUrU6cofY9/qSIxs3l5VPjt47pMfwDAtU2xvl+dUGBTWNGDlZOVpYmTH5PdS7yvXer1V9tY2K1YukS9e470PwmsfzzxSqw2dQ5rmuFzu+OAgID1b6D87MbBTabhiQMUFbWEb37/ixFRta/6Gv8sGG9nhr9uNq0jda4V8Y7fZXVnXhcFNZvO6AFX2zU80PvVs0Qq/al5ajXXTGKrBWqQc/NdfV4HuWbpDmSpJyMA5Kkbd+sUNrubZKkTj17SZLqRjl/86KGpeyNqtYNUYqK7lgZo7q9cWOf0q7tP+qOu3rqwP4UHdh/7nvyvhaLOnWOM3/uO2iYhvbvrREJfdTjnvuVnZWp+fNmKTr2FsV06OSK8d3SmKdGaduPW3VPz/uUmrLP4XkDX4ufbouLlyQdOpSh4UMTVKVKFcV37ablyz53eJ3GTaLUJKpppc7+W3hcFCSp379m6ZnBPfTQnTEKDrBoW3KG7h0+Rd9udK+HSK52P38qecuqpeafz0YBV8be5N2SpM8XJ+nzxUkOx66vVdshCk2aNtf4N9/VtLcnavKkV2Wx+OmOu3qq/+ARlTmy29u9q+zvSluUtECLkhY4HKtVO8JEISM9Xfkny76M8fKLzk/2D0wY4lZRqFJaWlpanhN9Ww+p6Flwninv/t3VI1xz4htd7+oRrikhfs5fm0XF8i3HN/Ld98YXAOCKIwoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMKqUlpaWlufEmRvSKnoWnKdNeLCrR7jmhFq9XT3CNeWncr3z4EqqE3z5f8f5TQEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBR3dUDXAn5x49q/bKFyti7S5mpe1R0ukB/+cd4RTa/+ZLXnT6VrymjHpXtRK56DhurZrG3Vs7AHuD40Wx9tvBDJe/cpn17dup0gU3PvjZVLW6Odjr3p59+0vLPFmr54gXKzEiTTw1fNWjcVPf3fkxNb2zlgundzw/r12rF0s+0bcsmZWcdUUhoqFpHx6rvwCEKDavpdP62rZs19c0JSt69UxY/P3WJ76b+CcPla7G4YHr3tHH9Wq1c9pl+3LJJOWf2/Oa2sepzkT3fvnWzpr113p7HdVM/N9xzj4jC0cNpWrP4Y4WER6hm3QbKSN5RrutWJ86QvfB0BU/nmTLSDmjRRzNVK6Ke6jVopD07tl703NlTJ2lx4lzdGv8Hdbv7AZ3KP6nlSxbomZH99eIb76tx0xaVOLl7mvbWRJ08kafOcV1Vp26kDmWka9H8D7Xmm1WaPidRIaFh5ty9e3bpiSGPKbL+DRo8YrSyszL18dyZykg7oFcmTXHhT+Fe3n27bM9vva2rIupG6vChdH06/0Ot/XaVps123vNRQx9TvcgblDC8bM8/mTdT6WkH9G8323OPiEJ4gyYaOXWhfK0B2rlutZKSn7/sNVlpqdq4crE69eyt1YkzKn5ID9OwSTN9kPSl/AMCtWbVCr32/IWjUFJSrGWLE9X+1jgNG/OCWe/QOV5/63W3/m/F50ShHAaPGK2WrdqoatVzd3xjOnTUiEF9lDR/nvoNGmbWp09+Xf7+AZo4+X35Wa2SpPBaERo/7lmtX/ud2rW/pdLnd0cJw0erxc/3vH1HjUzoo0Xz56nveXv+3jtlez7hnffl51e259fXitCEl5/VhnXfKTrWffbcIz5T8PG1yNca8IuuWT57sppEd1TdKN6Qfg1fi5/8AwIve15xcbGKCgsVFBzqsB4YFKKqVavK28enokb0KK1aRzu8OZ1dCwgI1IHUVLN2Kj9fG75fq/juPUwQJKnrH+6Wr8Wir1cuq7SZ3d1NF9jzm1pHyz8gUAf3n7fnp/L1w/drFdethwmCdN6er3CvPfeI3xR+qZ3rViljz3YN+M/7ysvOdPU4Hs3Hp4YaN2uhr5ctVpPmLdWsZWudys9X4px35Wf11+133uvqEd1Wgc2mggKbAoOCzFrKvmSVlBQrqllzh3O9vLzUqHFTJe/eWclTepYCm02nC2wKOG/PU/defM8bNm6qvXvca8+vuSjYiwq1cu5Uxdxxn4JqhhOFSjBszIua+MJTeuPlf5m162tF6MU33tf1teu4cDL3lvjRbNntdv3+9u5m7VhOtiQpNNT5g9CQsDD9uHljpc3niRac3fP4c3t+9GjZnodc4MPnUDfc82suCmv++5F+KinRLX982NWjXDN8fS2qU7+hmjS/SS3btFPusaNK+nCGXh37hF6YNF0BgcGuHtHtbNm0QTOnT1GX+G5qEx1r1gsLCyVJXt7eTtd4e/uY4/jltm7aoFnvTVGXuG5qfd6eF53ZU28vz9hzt4pCSbFdBfknHdYsAYGqWrVaua7Pzc7U2s8+UbdHh8q7hm9FjOhx7Ha78k/mOawFBAarWrXy7XlJSbGefzJBN7aKVr+hT5r1m9rEamS/B/Tpx7PVe8CwS7zCtcVut+vkCcf9Dgxy3O+D+1M09skRatCwkUY//ZzDuT5nPqOxFxU5vXZRUaE5jnPKved/L9vzJ36252c/Fyuye8aeu1UU0vds19yXRjmsDZ40R0E1w8t1/erEGfIPDlNks1bKPXPbKD/vuCTJdjJXudmZCgy9TlWqesTn71fE7u1b9OwTAx3WJs9drOvCa5fr+h1bN+pg6j79ddDjDuu16tRTRL0G2r1985Ua1SNs37pZIwf3dVj7MGmpwmtHSJKyjmRq9LCB8rNa9e+Jk2Xx83M49+wtjLO3NM53LCfngt+vv9Zt37pZT/zNcc/nLnTc878PHyir1apxE5z3/OyturO37s531A333K2icF1kQz005hWHNWtgSLmvP3E0S8ePZGjyyN5Ox5Z98IYk6fFpi1TjvG8QXOvqN2yisa9OdlgLCgm9yNnO8o4fk1T2ANvPlRQXq6Sk5LcN6GEaNm6i8W9Oc1g7+334vLxcjR42QPaiIr02bdYF32waNGykatWqa/fOHQ73ve12u/Ym71KXuG4V+wO4oYaNm+jVNy6+538fNkBFRUV6feqF97z+eXve5Wd7vi95lzq72Z67VRR8/fzVoEXbX3195wf6yHbyhMNadlqqVifOUPsef1JE4+by8qnxW8f0KFb/AN3UNvbyJ15ErTqRkqRvv1qm1jHnvqudsmenDqUfUPydPX/zjJ7EPyBQbWM6OK0XFNj01MjBysnK0oTJ76lOvcgLXm+1+qttTKxWLF2iR/oONP9V+8Xni1Vgs6lLXNcKnd8dXWrPnx45WDnZWXrtMnvepl2sVi5bot7n7fnyM3ve+Tb32nO3isKlfJM0R5KUk3FAkrTtmxVK271NktSpZy9JUt2olk7X1bCU/Q9Y64YoRUV3rIxRPUbinOmSpLT9KZKk1cv/p13bNkuS7u/1mKSyh9xuahurr79YIpvtlFpFt1fu0Rx9vuhjeXv76M77+MC/PF4a+5R2bf9Rd9zVUwf3p+jgmT2XJF+LRZ06x5k/9xs0TEP699aIhD7qcc/9Z56unaXo2FsU06GTK8Z3S+OeeUq7dvyo7nf11IHUFB1Ivfie9x00TMMG9NbI8/Z8/ofuuedVSktLS8tz4swNaRU9y28y7i/xFz329NwVFz12YMdmzX1p1FX3dx+1Cb/6v5Fzf9zFf2tLXPmD+efCwtP67yez9e1XXygrM0PVq3upWcvW+nOfBDVoFFUZo5ZLqNX52yNXiz/f001HDh+64LHra9XWR4scH5D6cfNGTX17Ytnfw2PxU5e4ruo/eITT/XBX+qlc7zyu8/A93XQk8yJ7Hl5b8y6w5+++PVHJe8r2vHNcVz2WcHXteZ3gy/877jFR8DTuEAVPczVHwRNd7VHwROWJAl+zAQAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBRpbS0tLQ8J2bkFlX0LIBLHc/n3/HKtOHwMVePcM15tF29y57DbwoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAACM6q4e4ErYuH6tViz9TNu2bFJ21hGFhIaqdXSs+gwcotCwmk7nb9u6WdPenKDk3Ttl8fNTl/hueixhuHwtFhdM757Y88p1/Gi2liz8UMk7t2nfnp06XWDTc69NVYubo53O/emnn7T8s4X6YvECZWakyaeGr25o3FT3935MTW9s5YLp3VP+8aNavyxJh/btUmbqHhWdLtDDT49XZPNL7+HpU/maOrqPbCdy1XPYv9Q05tZKmvjK8IgoTHtrok6eyFPnuK6KqBupwxnpWjT/Q635ZpXenZOokNAwc+7ePbs0ashjqlf/BiWMGK3srEx9MnemMtIO6N+Tprjwp3Av7Hnlykg7oEUfzVStiHqKbNBIu3dsvei5s6ZO0uLEubo1/g/qdvcDsuWf1BdLFmjsyP566Y331bhpi0qc3H0dPZyutUs+VnB4hGrWbaCM5B3lum71gpmyF56u4OkqjkdEIWHEaLVs1UZVq567G9auQ0eNHNRHSfPnqd+gYWZ9+uTX5e8foImT35ef1SpJCq8VodfGPav1a79Tu/a3VPr87og9r1wNmzTTjKQv5R8QqDWrVmj38xeOQklJsZYtTlSHW+M0fMwLZr1D53gN7nW3Vq/4nCiUU3iDxhoxZYF8rQHa9f1qJZUjCtlpqdq0crE63tNL/7dgZiVMeeV5xGcKrVpHO7w5nV0LCAjUwdRUs3YqP18/fL9W8d17mDcnSer6h7vla7Fo1cpllTazu2PPK5evxU/+AYGXPa+4uFhFhYUKDA51WA8MClHVqlXl7eNTUSN6HB9fi3ytAb/omuWzJ6tJ246qG9WygqaqeB7xm8KFFNhsKiiwKTAoyKyl7ktWSUmxopo1dzjXy8tLjRo3VfLunZU8pWdhz13Px6eGGjdroa+XLVZU85Zq1rK1TuXnK3HOu/Kz+uv2O+919Ygea+e6VcpI3qH+r76nvOwjrh7nV/PYKCR+NFt2u11dbu9u1o7mZEuSQkKdPwgNCQvTj5s3Vtp8nog9vzoMH/OiJrzwlF5/+V9m7fpaEXrpjfcVXruOCyfzXPaiQn05b5radb9XQTXDicLVZsumDZo1fYq6xHdTm+hYs15YWChJ8vL2drrG29vHHMcvx55fPXx9Lapbv6GaNL9JN7Vpp+PHjirpwxl6ZewTenHSdAUEBrt6RI+zZvFH+qmkWLf88WFXj/KbuVUU7Ha7Tp7Ic1gLDApWtWrVzJ8P7k/RM0+OUIOGjTTq6ecczvU5cz/VXlTk9NpFRYXmOM5hzyuX3W5X/knH/Q4IdNzvSykpKdZzTyboxlbRemzok2b9pjaxGtnvAX368Wz1HjDsEq9w7Skptqsg/6TDmiUgUFWrlm/Pc7Mzte6z+er21yHyruFbESNWKreKwvatm/X44L4Oa/OSliq8doQkKetIpp4cNlB+VqtenjhZFj8/h3PPfn/+2NFsp9c+lpNzwe/XX+vY88q1e/sWPfPEQIe1d+Yu1nXhtct1/Y6tG3UwdZ8eHfS4w3rtOvUUUa+Bdm3ffKVG9Rjpe3Zo3rhRDmsJE2crqGZ4ua7/vwUz5R8cqnrNWik3O1OSdCrvmCTJdiJPudmZCgy9TlWqusf3etwqCg0bN9F/3pzmsHb2+/B5ebl6ctgA2YuKNH7arAu+2dRv2EjVqlXX7p071CX+3H1vu92uvcm71CWuW8X+AG6IPa9c9Rs20dhXJzusBYWEXuRsZ7nHy96MfvrpJ6djJcXFKikp+W0DeqDrIm/Qn596xWHNGhhS7utP5GTp+JFDeufxR5yOLZvxhiRp5NQk1fCzOh2/GrlVFPwDAtU2poPTekGBTWNGDlZOVpYmTH5PdepFXvB6q9VfbWNitWLpEvXuO9D8V+3yzxerwGZT57iuFTq/O2LPK5fVP0Ct2sZe/sSLqF2n7H+Hb75aptYx557/SNmzU4fSDyj+zp6/eUZP4+vnrwYt2vzq6299oI8KfnbLLzt9v1YnzlD7Hg8qolFzefnU+K1jVhq3isLFjBv7lHZt/1F33NVTB/an6MD+FHPM12JRp85x5s99Bw3T0P69NSKhj3rcc7+yszI1f94sRcfeopgOnVwxvltizytf4pzpkso+w5GkVcv/p13bNkuS7u/1mKSyh9xatY3V118sUYHtlFpFt9fxozn6fNHH8vb2UY/73P+D0Mr07aK5ksre5CVp27crlL5nmySp4z1/kSTVjXJ+GNDHUvZbQa0botQkumMlTHrlVCktLS0tz4kZuc4fFF4tHrqnm44cPnTBY9fXqq0PFzk+IPXj5o2a9vbEsr+Hx+KnznFd1X/wCKf74bg4T9zz4/lX77/jknRfXNuLHluw8gfzz4WFp/XfT2brm6++UFZmhqpX91Kzlq31UJ8ENWgUVRmjlsuGw8dcPcJlvdzr9oseGzNn+UWPHdixRfPGjbrq/u6jR9vVu+w5HhEF4Eq42qPgadwhCp6mPFFwj4/DAQCVgigAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwKhSWlpaWp4TTxdX9Cg437H8IlePcM1ZsfeIq0e4pgzs/4qrR7jmFGx667Ln8JsCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwCAKAACDKAAADKIAADCIAgDAIAoAAIMoAAAMogAAMIgCAMCo7uoBroR1a9fosyX/1aaNG5V1JFOhYWGKiW2vvw0drpo1rzPnFRQU6NOkhfrqq5Xau2ePbLZTqlsvUvc/8KDue+BPqlatmgt/Cvfyw/q1WrH0M23bsknZWUcUEhqq1tGx6jtwiELDajqdv23rZk19c4KSd++Uxc9PXeK7qX/CcPlaLC6Y3v2cPH5U65ct1KG9u3Q4dY+KTheo1z/GK7L5zZe87vSpfL0z6lHZTuTq3mFj1Sz21soZ2MN4e1XX2IQ79XCPGAX5+2pb8iE9+/YSfblul6tHu+I8IgqTJvxHeXl5ur1rd0VG1ld6epo+mjdHq7/+Wp8sWKSwmmVvUunpafr3uBcU276Dev/1UflZrfru22/00gvPaeuWLXrx5Vdc/JO4j2lvTdTJE3nqHNdVdepG6lBGuhbN/1Brvlml6XMSFRIaZs7du2eXnhjymCLr36DBI0YrOytTH8+dqYy0A3pl0hQX/hTu49jhNK1Z/LFCwiNUs24DZSTvKNd1qxJnyF54uoKn83zvPt9LPeNa6615X2nvwWz1vjtWi95MUPcBr+u7zSmuHu+K8ogojHpyjFq3aauqVc/dDevY6Xfq+9de+mjeHA0ZPlKSFBYWpsRFi9WoUWNz3gMP/llj/zlGnyYt1IBBg1UvMrLS53dHg0eMVstWbRz2PKZDR40Y1EdJ8+ep36BhZn365Nfl7x+giZPfl5/VKkkKrxWh8eOe1fq136ld+1sqfX53E96giR6fulC+1gDtXLdaC5Ofv+w1WWmp2rhysTr17K3ViTMqfkgPFX1jpB7sHq0xE5I0afZKSdLcJev0w/x/6KUR9+j3j05w8YRXlkd8ptA2up3Dm9PZtcDAIKWknKt4cHCIQxDOui3udklSSsq+ih3Ug7RqHe20561aRysgIFAHUlPN2qn8fG34fq3iu/cwQZCkrn+4W74Wi75euazSZnZnPr4W+VoDftE1y2dPVlR0R9WLalFBU10besbfrOLiEr238FuzVlhUrBmfrlH7VjeozvVBrhuuAnhEFC7EduqUbLZTCgoOvuy5R3NyJEnB5TgXF1dgs6mgwKbAoCCzlrIvWSUlxYpq1tzhXC8vLzVq3FTJu3dW8pTXhp3rVil9z3bd9tAAV4/i9lo1ravkg1k6ecrxNtyGbfslSTdF1XHBVBXHY6MwZ/ZM2e12det+xyXPsxcVac7smYqoU0c3tmhZSdN5psSPZstut+v3t3c3a8dysiVJoaHOHz6HhIXp6JnjuHLsRYVaMXeqYu64T0E1w109jtsLDwtQZvYJp/XMnLK1WjUDK3ukCuURnyn83A8b1mvqO2+ra/c7FNu+wyXPffmlF5Syb6/eemeaqlf3yO2oFFs2bdDM6VPUJb6b2kTHmvXCwkJJkpe3t9M13t4+5jiunO/++5F+KilRxz8+7OpRPIKvj5cK7cVO66cL7ea4J3Grd0F7UZHy8vIc1oJDQhy+Spqask8jhw1Rw0aN9ezzL17y9Wa8P10LEj/R34YO1+9u7VwhM7s7u92ukycc9zwwKNhhzw/uT9HYJ0eoQcNGGv30cw7n+vj4lL1OUZHTaxcVFZrjKFNSbFdB/kmHNUtAoKpWLd/XpXOzM7X2s0/U/dGh8q7hWxEjXnMKCu3y8XJ+q6xxJgYFZ+LgKdwqCps3b9JjfR5xWPvfFysVEVF2Ty/z8GEN6t9PVn+r3p4yTX5+1gu9jCTp06SFmjRhvB740581YNDgCp3bnW3fulkjB/d1WPswaanCa0dIkrKOZGr0sIHys1r174mTZfHzczg35MwzC0ePOt8mOpaTc8FnGq5l6Xu2a85LoxzW/jZpTrlvA61OnCH/4DBFNmul3OxMSVJ+3nFJku1krnKzMxUYep2qVPXYO8dXXGbOCdW+zvkWUXhY2Qf/h7PznI65M7eKQlRUU02d/oHDWtiZN5Xc3OMaNKCviuxFmvH+PIeH1n7uqy9X6Lln/qm4+K56+p/PVOjM7q5h4yYa/+Y0h7WzzyDk5eVq9LABshcV6bVpsy74Bt+gYSNVq1Zdu3fu0O/jz33WYLfbtTd5l7rEdavYH8DNXBfZUA+PcXxexhoYUu7r845m6fiRDL09srfTsaUfvCFJemLaItW4xH8wwdHW3enqHN1Y/n41HD5sbteivjnuSdwqCgGBgWrfwfk77TabTX8bNEBZR45o+gezFBlZ/6Kv8cOG9fr7qMfVpm20Xn51vNPXKuHIPyBQbWOcP5cpKLDpqZGDlZOVpQmT31Odehd+vsNq9VfbmFitWLpEj/QdaH6T+OLzxSqw2dQlrmuFzu9ufP381aBF2199fZcH+sh20vFD0ey0VK1KnKEOPf6kiMbN5eVT47eOeU1JWrFJI/8ar373djTPKXh7Vdcjf2yv77emKv1IrmsHvMLcKgoX8/TfR2nbj1t1z733KSVln8PzBhaLn26Li5ckHTqUoeFDElSlShXd3rWbvlj2ucPrNGkSpSZRTSt1dnf10tintGv7j7rjrp46uD9FB/efex7E12JRp85x5s/9Bg3TkP69NSKhj3rcc7+yszL1ybxZio69RTEdOrlifLf0TdIcSVJ2xgFJ0o/frFDa7m2SpE49e0mS6kY5f4OuhqUsxLVuiFJUdMfKGNWjrN92QAu+2Kjnh96tmiFW7UvLUa+7YhRZK1SDnpvr6vGuOI+Iwu5dZX//yKKFC7Ro4QKHY7VrR5goZKSn6+TJsg/xxr3o/ETooMFDiEI57U3eLUn6fHGSPl+c5HDs+lq1HaLQpGlzvfbmu5r69kS9PelVWSx++sNdPdV/8IjKHNntrfrZU8lbVi01/3w2CqgY/f41S88M7qGH7oxRcIBF25IzdO/wKfp2o+c98FqltLS0tDwnnnb+RhYq0LF852/roGKt2HvE1SNcUwb25+8aq2wFm9667DncUAcAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAAZRAAAYRAEAYBAFAIBBFAAABlEAABhEAQBgEAUAgEEUAAAGUQAAGEQBAGAQBQCAQRQAAEaV0tLSUlcPAQC4OvCbAgDAIAoAAIMoAAAMogAAMIgCAMAgCgAAgygAAAyiAAAwiAIAwPh/xMJlXX621UgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# run mc policy prediction for 100000 episodes for random policy\n",
    "V_pi = mc_policy_eval(\n",
    "    random_policy,\n",
    "    env,\n",
    "    discount_factor=1.0,\n",
    "    episode_count=100000)\n",
    "\n",
    "# Print policy\n",
    "grid_print(V_pi.reshape(env.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "We see that state values do not converge very well for 100 episode simulation. However, with 10,000 episode simulation the values have come close to those we saw in `3.a-policy-evaluation.ipynb` where policy evaluation using DP was carried out. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
